Obstacle avoidance apparatus and obstacle avoidance method

ABSTRACT

An obstacle detecting unit detects an obstacle for a user wearing a head mounted display from an image of the outside world. A distance calculating unit calculates the distance from a detected obstacle to the user wearing the head mounted display. An obstacle replacing unit replaces the detected obstacle with a virtual object. A virtual object synthesizing unit generates a virtual object at a position within a virtual space displayed on the head mounted display, in which the position is determined according to the distance to the obstacle.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an apparatus and a method for obstacleavoidance used for a head mounted display.

2. Description of the Related Art

It is common that a user plays a game while wearing a head mounteddisplay connected to a game device on the head, watching the screendisplayed on the head mounted display, and operating a controller or thelike. With a typical stationary display connected to a game device,since the user's field of vision includes the outside of the display'sscreen, the user may not be able to concentrate on the screen or theuser's feeling of being immersed in the game may be reduced. Wearing ahead mounted display, on the other hand, the user only sees imagesdisplayed on the head mounted display, so that the user feels moreimmersed in the world of the images and the entertainment capabilitiesof the game will be enhanced.

However, since a user wearing a head mounted display cannot see theoutside world, the user may bump into an obstacle around the user or mayknock something off a desk when stretching a hand or moving the user'sbody during game play.

SUMMARY OF THE INVENTION

The present invention has been made in view of such a problem, and apurpose thereof is to provide an obstacle avoidance apparatus and anobstacle avoidance method that can avoid the situation where a userwearing a head mounted display bumps into an obstacle.

To solve the problem above, an obstacle avoidance apparatus of anembodiment of the present invention comprises: an obstacle detectingunit configured to detect an obstacle for a user wearing a head mounteddisplay from an image of the outside world; an obstacle replacing unitconfigured to replace the detected obstacle with a virtual object; and avirtual object synthesizing unit configured to generate the virtualobject within a virtual space displayed on the head mounted display.

Another embodiment of the present invention relates to an obstacleavoidance method. The method comprises: detecting an obstacle for a userwearing a head mounted display from an image of the outside world;replacing the detected obstacle with a virtual object; and generatingthe virtual object within a virtual space displayed on the head mounteddisplay.

Optional combinations of the aforementioned constituting elements, andimplementations of the invention in the form of methods, apparatuses,systems, computer programs, data structures, and recording media mayalso be practiced as additional modes of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments will now be described, by way of example only, withreference to the accompanying drawings which are meant to be exemplary,not limiting, and wherein like elements are numbered alike in severalFigures, in which:

FIG. 1 is an outline view of a head mounted display;

FIG. 2 is a functional configuration diagram of a head mounted display;

FIG. 3 is a configuration diagram of an obstacle avoidance systemaccording to the present embodiment;

FIG. 4 is a diagram for describing an environment around a user wearinga head mounted display;

FIG. 5 is a functional configuration diagram of an obstacle avoidanceapparatus according to the present embodiment;

FIG. 6 is a flowchart that shows a procedure of obstacle avoidanceperformed by the obstacle avoidance apparatus shown in FIG. 5;

FIG. 7 is a diagram for describing an exemplary virtual object displayedon a head mounted display to avoid an obstacle; and

FIG. 8 is a diagram for describing another exemplary virtual objectdisplayed on a head mounted display to avoid an obstacle.

DETAILED DESCRIPTION OF THE INVENTION

The invention will now be described by reference to the preferredembodiments. This does not intend to limit the scope of the presentinvention, but to exemplify the invention.

FIG. 1 is an outline view of a head mounted display 100. The headmounted display 100 comprises a main body 110, a frontal region contactunit 120, and a temporal region contact unit 130.

The head mounted display 100 is a display apparatus that is mounted onthe head of a user and used to view a still image or a moving imagedisplayed thereon or to listen to sound or music output through theheadphones.

Position information of the user can be measured using a positionsensor, such as a global positioning system (GPS), embedded in the headmounted display 100 or placed externally. Also, posture information,such as the orientation or tilt of the user's head wearing the headmounted display 100, can be measured using a posture sensor embedded inthe head mounted display 100 or placed externally.

The main body 110 includes a display, a position information acquisitionsensor, a posture sensor, and a communication apparatus. The frontalregion contact unit 120 and temporal region contact unit 130 include abiological information acquisition sensor that can measure the user'sbiological information, such as the body temperature, pulse, bloodcomponents, perspiration, brain waves, and cerebral blood flow.

The head mounted display 100 may further comprise a camera for capturingan image of a user's eye. With the camera mounted on the head mounteddisplay 100, the line of sight, pupil movement, and blinking of the usercan be detected.

There will now be described an obstacle avoidance method applied when auser wears the head mounted display 100. The application of the obstacleavoidance method of the present embodiment is not limited to the headmounted display 100 in the narrow sense and is also available toeyeglasses, a glasses-type display, a glasses-type camera, headphones, aheadset (headphones with a microphone), earphones, earrings, a ear-hookcamera, a headwear, a but with a camera, or a headband.

FIG. 2 is a functional configuration diagram of the head mounted display100.

A control unit 10 is a main processor that processes and outputs asignal including an image signal and a sensor signal, an instruction,and data. An input interface 20 receives an operation signal or asetting signal from a touch panel and a touch panel controller andprovides the signal to the control unit 10. An output interface 30receives an image signal from the control unit 10 and displays the imageon the display. A backlight 32 provides backlight to the liquid crystaldisplay.

A communication control unit 40 transmits data input from the controlunit 10 to the outside through a network adapter 42 or an antenna 44 viawired or wireless communication. The communication control unit 40 alsoreceives data from the outside through the network adapter 42 or antenna44 via wired or wireless communication and outputs the data to thecontrol unit 10.

A storage unit 50 temporarily stores data, a parameter, an operationsignal, and the like processed by the control unit 10.

A GPS unit 60 receives position information from a GPS satellite andprovides the information to the control unit 10 according to anoperation signal from the control unit 10. Also, a wireless unit 62receives position information from a wireless base station and providesthe information to the control unit 10 according to an operation signalfrom the control unit 10.

A posture sensor 64 detects posture information including theorientation and tilt of the main body 110 of the head mounted display100. The posture sensor 64 is implemented by combining a gyro sensor, anacceleration sensor, an angular acceleration sensor, and the like, asneeded.

An external input/output terminal interface 70 is an interface throughwhich a universal serial bus (USB) controller or another peripheraldevice is connected. An external memory 72 is an external memory such asa flash memory.

A clock unit 80 sets time information according to a setting signal fromthe control unit 10 and provides time data to the control unit 10.

A camera unit 90 includes configurations necessary for shooting, such asa lens, an image sensor, and a range sensor. A range sensor is a sensorthat emits infrared light and shoots the reflected light so as tomeasure the distance to the object based on the principle oftriangulation.

The control unit 10 can provide an image or text data to the outputinterface 30 to display the image or data on the display or can providesuch an image or text data to the communication control unit 40 totransmit the image or data to the outside.

FIG. 3 is a configuration diagram of an obstacle avoidance systemaccording to the present embodiment.

Head mounted displays 100 a and 100 b of multiple users are connected toa game device 300 a through an interface provided to connect aperipheral device with a wireless communication function or a USBperipheral device to the game device 300 a. To another game device 300 bis connected a head mounted display 100 c of another single user.

The game devices 300 a and 300 b (also collectively referred to as gamedevices 300) and a server 200 are connected to a network 400. The gamedevices 300 receive content including a game and a movie from the server200 via the network 400. The server 200 may also provide to the gamedevices 300 an online application, such as a game in which multipleusers can participate via the network 400.

FIG. 4 is a diagram for describing an environment around a user wearingthe head mounted display 100. The user operates a controller 310 of agame device 300 while wearing the head mounted display 100. Some gameapplications require a user holding the controller 310 to move a hand orthe body of the user. However, since a user wearing the head mounteddisplay 100 cannot see the outside world, the user may bump into a table500 or a chair 510 and might knock something off the table 500 or getinjured.

In the case where the head mounted display 100 is a see-through headmounted display through which the user can also see the outside worldbesides images, the user can check the surroundings. However, when theuser is focused on what is displayed on the screen, the user may not beable to pay enough attention to an obstacle around the user, and hence,the same kind of problem will arise.

In the present embodiment, a real obstacle around the user is detected,replaced by a virtual object, and displayed on the screen of the headmounted display 100, so that the user can naturally avoid the obstacle.

FIG. 5 is a functional configuration diagram of an obstacle avoidanceapparatus 600 according to the present embodiment. FIG. 5 is a blockdiagram featuring the functions, and these functional blocks may beimplemented in a variety of forms by hardware only, software only, or acombination thereof.

The obstacle avoidance apparatus 600 is basically mounted on a gamedevice 300 to which the head mounted display 100 is connected, but atleast part of the functions of the obstacle avoidance apparatus 600 maybe implemented by the control unit 10 in the head mounted display 100.Alternatively, at least part of the functions of the obstacle avoidanceapparatus 600 may be implemented by the server 200.

An obstacle detecting unit 610 detects an obstacle from an image of theoutside world captured by a camera 140 mounted on the head mounteddisplay 100. For the detection of an obstacle, a conventional algorithmfor extracting an object from an image is used. By shooting a movingimage, the movement of an object can be detected, so that movableobjects, such as humans and animals, can be distinguished fromstationary objects, such as desks and ornaments. When a camera is notmounted on the head mounted display 100, another camera connected to agame device 300 may be used to capture an image of the outside world soas to detect an obstacle. Also, when an obstacle is to be detected usingthe camera 140 mounted on the head mounted display 100, obstacles arounda user wearing the head mounted display 100 cannot be detected if theuser only looks to the front. Accordingly, a message for leading theuser to turn the user's head to look around may be displayed on the headmounted display 100.

A distance calculating unit 620 calculates the distance from an obstacledetected by the obstacle detecting unit 610 to the user. The distancefrom the user to an obstacle can be measured by a range sensor providedin the camera unit 90.

When a user wearing the head mounted display 100 moves, the positionalrelationship between the user and an obstacle changes. Accordingly, witha predetermined period while the user is wearing the head mounteddisplay 100, the obstacle detecting unit 610 detects an obstacle from acaptured image of the outside world and the distance calculating unit620 measures the distance to the obstacle.

An obstacle replacing unit 630 replaces an obstacle detected by theobstacle detecting unit 610 with a virtual object to be displayed in avirtual space in a game or another application. The size of the virtualobject, which is generally determined so as to correspond to the size ofthe obstacle shot by the camera 140, may be determined according to thedistance to the obstacle calculated by the distance calculating unit620, so that an obstacle located closer to the user may be displayed asa larger virtual object, for example.

A virtual object synthesizing unit 640 renders a virtual object, withwhich the obstacle replacing unit 630 has replaced an obstacle, within avirtual space in a game or the like and displays the composite image onthe head mounted display 100. The virtual space display unit 640transmits the composite image to the head mounted display 100 viawireless communication or via a USB cable or the like.

The virtual object synthesizing unit 640 may generate a composite imageby superimposing a virtual object upon a background image, at a positionin a virtual space determined according to the distance to the obstacleacquired by the distance calculating unit 620.

The virtual object is an object in a virtual space to which a userhesitates to get closer, such as a fence, a cliff, a sandbag, a box, anda no-entry sign. It is preferable to select a virtual object that is notunnatural and coordinates with the landscape of the virtual space. Byarranging such a virtual object in a virtual space, a user can naturallyavoid bumping into a real obstacle corresponding to the virtual objectwhile concentrating on the application such as a game.

The distance calculating unit 620 calculates the distance between theuser and an obstacle in real time and, when the distance to an obstacleis less than a predetermined threshold, a warning unit 650 warns theuser not to get any closer to the obstacle by allowing the head mounteddisplay 100 to sound an alert or allowing the controller 310 of the gamedevice 300 to vibrate.

FIG. 6 is a flowchart that shows a procedure of obstacle avoidanceperformed by the obstacle avoidance apparatus 600. FIGS. 7 and 8 arediagrams for describing exemplary virtual objects displayed on the headmounted display 100 to avoid obstacles.

Information on the user's surroundings is acquired using a camera or adistance measuring sensor mounted on the head mounted display 100 wornby the user (S10). The obstacle detecting unit 610 then analyzes theacquired image data to detect an obstacle (S12). Also, the distancecalculating unit 620 uses depth information of the object acquired bythe distance measuring sensor to calculate the distance between the userand the obstacle (S14).

The obstacle replacing unit 630 determines the type of a virtual objectby which the obstacle detected by the obstacle detecting unit 610 is tobe replaced (S16). The virtual object is an object that is not unnaturalin the content displayed on the head mounted display 100 and that a usertries to avoid bumping into. The virtual object synthesizing unit 640generates the virtual object within the virtual space and displays thecomposite image on the head mounted display 100 (S18).

In the example of FIG. 7, a user wearing the head mounted display 100 isplaying a shooting game, using the controller 310 as a rifle. If thereis an obstacle right in front of the user's body, a fence 520 as avirtual object replacing the obstacle will be displayed right in frontof the user's rifle within the virtual space in the game so that theuser will not thrust the controller 310 forward or move the user's bodyforward. Accordingly, the user will naturally operate the controller 310corresponding to a rifle before the fence 520, so as to avoid operatingthe controller 310 beyond the fence 520, or the obstacle.

In the example of FIG. 8, a user wearing the head mounted display 100 isplaying a role-playing game, using the controller 310 to operate acharacter and exploring the virtual space. If there is an obstacle aheadof the user on the right, a collapsed cliff 530 will be displayed as avirtual object on the path leading toward the right and a virtual signsaying “Danger” will also be displayed before the path so that the userwill not thrust a hand or move the body forward toward the right.

If the game is a sports game, such as a bowling game, a tennis game, anda table tennis game, the playing area may be restricted or a foul linemay be provided according to the position of an obstacle. Also, the gameapplication may control the game play so that the tennis ball or thelike does not reach to a position corresponding to the position of areal obstacle, thereby preventing the user from bumping into theobstacle. In some board games, a user wearing the head mounted display100 himself or herself moves as a game piece. In such a case, the gameapplication may create the squares of the board game so that any squareis not arranged in a position corresponding to the position of a realobstacle, thereby preventing the user from bumping into the obstacle.

When the obstacle is a movable object such as a human or an animal, theobstacle replacing unit 630 selects a movable object, such as an avataror a robot, as a virtual object by which the obstacle is replaced, andthe virtual object synthesizing unit 640 generates within the virtualspace a movable object that moves according to the movement of the realobstacle.

The warning unit 650 determines whether or not the distance between theuser and an obstacle is less than a threshold (S20) and, if the distanceis less than the threshold (Y at S20), the warning unit 650 will warnthe user with sound issued by the head mounted display 100, a textdisplayed on the head mounted display 100, the vibration of thecontroller 310, and the like (S22). In a game, a warning against gettingcloser to an obstacle may be indirectly given to the user by providing adisadvantage in the game, such as the score being reduced or thecharacter being endangered when the character gets closer to a virtualobject provided as an obstacle.

The present invention has been described with reference to theembodiment. The embodiment is intended to be illustrative only, and itwill be obvious to those skilled in the art that various modificationsto constituting elements or processes could be developed and that suchmodifications also fall within the scope of the present invention.

What is claimed is:
 1. An obstacle avoidance apparatus, comprising:circuitry configured to determine a user's position as the user movesthroughout a real space; circuitry configured to generate a fictitious,virtual space that a user views through a head mounted display (HMD),where the virtual space includes images of landscape, scenery, andobjects of a fictitious space based, at least in part, on the user'sposition acquired as the user moves throughout the real space, and doesnot include images of landscape, scenery, or objects of a real spacewithin which the user is located; circuitry configured to detect a realobstacle from an image of the real space within which the user islocated; circuitry configured to generate a virtual object in responseto the detection of the real obstacle by generating the virtual objectin such a way as to be: (i) consistent with context established by thelandscape, scenery, and objects of the fictitious, virtual space thatthe user is experiencing through the HMD, and (ii) a different kind ofobject than, and is not recognizable by the user as representing, thereal obstacle of the real space within which the user is located; andcircuitry configured to locate the virtual object within the fictitious,virtual space displayed on the HMD.
 2. The obstacle avoidance apparatusaccording to claim 1, wherein the image of the outside world is capturedby a camera mounted on the HMD.
 3. The obstacle avoidance apparatusaccording to claim 1, wherein the circuitry configured to generate thevirtual object such that the user wearing the HMD is expected to avoidcontact with the virtual object in the fictitious virtual space, therebyavoiding contact with the real obstacle in the real space within whichthe user is located.
 4. The obstacle avoidance apparatus according toclaim 3, wherein, when the detected real obstacle is a movable object,the obstacle generating unit selects, as the virtual object, an objectthat moves within the fictitious, virtual space displayed on the HMD. 5.The obstacle avoidance apparatus according to claim 1, furthercomprising circuitry configured to calculate the distance from thedetected real obstacle to the user wearing the HMD, wherein thecircuitry configured to locate displays the virtual object at a positionwithin the fictitious, virtual space determined according to thedistance from the real obstacle to the user wearing the head mounteddisplay.
 6. The obstacle avoidance apparatus according to claim 5,further comprising circuitry configured to warn the user when thedistance to an obstacle calculated by the distance calculating unit isless than a predetermined threshold.
 7. An obstacle avoidance method,comprising: determining a user's position as the user moves throughout areal space; generating a fictitious, virtual space that a user viewsthrough a head mounted display (HMD), where the virtual space includesimages of landscape, scenery, and objects of a fictitious space based,at least in part, on the user's position acquired during the step ofdetermining the user's position as the user moves throughout the realspace, and does not include images of landscape, scenery, or objects ofa real space within which the user is located; detecting a real obstaclefrom an image of the real space within which the user is located;generating a virtual object in response to the detection of the realobstacle by generating the virtual object in such a way as to be: (i)consistent with context established by the landscape, scenery, andobjects of the fictitious, virtual space that the user is experiencingthrough the HMD, and (ii) a different kind of object than, and is notrecognizable by the user as representing, the real obstacle of the realspace within which the user is located; and locating the virtual objectwithin the fictitious, virtual space displayed on the HMD.
 8. Anon-transitory, computer readable storage medium containing an embeddedcomputer program, which when executed by a computer, causes the computerto carry out actions, comprising: determining a user's position as theuser moves throughout a real space; generating a fictitious, virtualspace that a user views through a head mounted display (HMD), where thevirtual space includes images of landscape, scenery, and objects of afictitious space based, at least in part, on the user's positionacquired during the determining the user's position as the user movesthroughout the real space, and does not include images of landscape,scenery, or objects of a real space within which the user is located;detecting a real obstacle from an image of the real space within whichthe user is located; generating a virtual object in response to thedetection of the real obstacle by generating the virtual object in sucha way as to be: (i) consistent with context established by thelandscape, scenery, and objects of the fictitious, virtual space thatthe user is experiencing through the HMD, and (ii) a different kind ofobject than, and is not recognizable by the user as representing, thereal obstacle of the real space within which the user is located; andlocating the virtual object within the fictitious, virtual spacedisplayed on the HMD.
 9. The obstacle avoidance method according toclaim 7, wherein the image of the outside world is captured by a cameramounted on the HMD.
 10. The obstacle avoidance method according to claim7, further comprising generating the virtual object such that the userwearing the HMD is expected to avoid contact with the virtual object inthe fictitious virtual space, thereby avoiding contact with the realobstacle in the real space within which the user is located.
 11. Theobstacle avoidance method according to claim 10, wherein, when thedetected real obstacle is a movable object, the method further comprisesselecting, as the virtual object, an object that moves within thefictitious, virtual space displayed on the HMD.
 12. The obstacleavoidance method according to claim 7, further comprising calculatingthe distance from the detected real obstacle to the user wearing theHMD; and displaying the virtual object at a position within thefictitious, virtual space determined according to the distance from thereal obstacle to the user wearing the head mounted display.
 13. Theobstacle avoidance method according to claim 12, further comprisingwarning the user when the distance to an obstacle calculated is lessthan a predetermined threshold.